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Abstract 

The  transition  to  the  digital  Army  of  Force  XXI  and  beyond  is  characterized  by  challenges  to 
how  the  Army  will  train,  maintain,  and  operate  as  an  information  age  force.  In  response  to  the 
concerns  and  issues  resulting  from  digitization,  the  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI),  Armored  Forces  Research  Unit,  is  engaged  in  the  design 
and  development  of  training  and  performance  evaluation  techniques  to  support  Force  XXI  digital 
capabilities.  This  paper  summarizes  an  ARI  project  that  addressed  training  for  leaders  and  staffs 
of  future  digital  environments. 

Based  on  learning  and  performance  research  on  team  training  and  operations  in  digital 
environments,  a  training  support  package  for  a  future-battlefield  experiment  was  constructed. 
Coordination  between  ARI  and  the  Mounted  Maneuver  Battlespace  Laboratory  (MMBL)  at  Fort 
Knox,  Kentucky  enabled  the  two  organizations  to  work  together  as  a  team  to  accomplish 
multiple  goals. 

Observations  and  data  collection  during  implementation  of  the  training  led  to  formulation  of  a 
series  of  lessons  learned  on  training  content  and  training  structure.  These  lessons  are  addressed 
to  several  audiences,  including  developers  of  future  staff  training  and  researchers  conducting 
future  systems  experiments  and  training. 


1  Statements  in  this  paper  are  those  of  the  authors  and  do  not  necessarily  reflect  the  official  opinions  or  policies  of 
the  U.S.  Department  of  the  Army. 
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Introduction 


The  transition  to  the  digital  Army  of  Force  XXI  and  beyond  is  characterized  by  challenges  to 
how  the  Army  will  train,  maintain,  and  operate  as  an  information  age  force.  The  foundation  of 
the  future  Army  includes  an  increased  area  of  operation;  enhancing  weapon  system  capabilities 
to  reduce  the  number  of  battlefield  systems;  non-linear,  asymmetrical  operations;  and 
information  dominance  with  certainty  for  situational  awareness.  The  future  force  will  depend  on 
the  capabilities  of  its  command,  control,  communication,  computer,  and  intelligence  (C4I) 
systems,  and  on  the  ability  of  the  commanders  and  staffs  to  use  the  systems. 

As  digital  information  systems  increase  the  amount  and  complexity  of  information  provided  to 
the  staff,  training  must  provide  the  higher-order  skills  needed  on  the  digital  battlefield.  “Better 
education  and  training,  devoted  to  information  processing  under  stress  and  in  environments 
characterized  by  uncertainty,  are  needed  to  develop  the  necessary  skills  to  handle  these 
information-rich  situations”  (Alberts,  1996,  p.  32). 

In  response  to  the  concerns  and  issues  resulting  from  digitization,  the  U.  S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  Armored  Forces  Research  Unit,  is 
engaged  in  the  design  and  development  of  training  and  performance  evaluation  techniques  to 
support  Force  XXI  digital  capabilities.  The  efforts  are  performed  in  support  of  the  ARI  Science 
and  Technology  Objective  (STO)  -  Force  XXI  Training  Strategies.  This  report  summarizes 
portions  of  an  ARI  project  that  addressed  methods  for  training  leaders  and  staffs  of  future  digital 
environments.  The  work  was  conducted  during  1998-1999.  It  began  with  a  thorough  review  of 
research  literature  and  technical  documentation  related  to  team  performance  and  training  and 
operations  in  digital  environments.  Based  on  the  literature  review,  a  training  program  was 
designed  and  developed  to  support  a  Mounted  Maneuver  Battlespace  Laboratory  (MMBL)  Battle 
Command  Reengineering  (BCR)  III  experiment  in  April  1999. 

This  paper  describes  the  development  work  and  the  prototype  products,  results  of  their  use 
during  the  MMBL  experiment,  and  implications  for  future  training,  both  within  MMBL  settings 
and  in  the  larger  context  of  staff  training. 

Research  and  Development  Method 

Researchers  focused  on  two  aspects  of  future  staff  training:  what  to  train,  and  how  to  train.  First, 
because  the  focus  was  on  staff  training  in  a  digital  environment,  the  literature  on  various  aspects 
of  team  performance  in  information-rich  environments  was  examined.  Second,  different 
characteristics  of  the  structures  of  training,  in  terms  of  the  order  of  presentation,  pace,  and 
delivery  mode,  were  examined  in  relation  to  the  content  areas.  The  review  led  to  specific  design 
decisions  concerning  content  and  structure  of  training  for  future  staffs. 

Training  Content 

Analysis  of  digital  environments  and  operations,  as  well  as  results  from  previous  MMBL 
experiments  (e.g.,  Elliott,  1998;  Elliott  et  al.,  1998),  indicated  that  the  content  of  the  training  for 
future  staffs  should  cover  three  training  needs:  system  operations,  tactical  skills  using  the 


capabilities  of  the  new  environment,  and  team  process  skills  to  bring  the  staff  quickly  to  a  high 
level  of  proficiency. 

•  System  operations  training :  The  initial  training  requirements  is  for  individual  training. 
Each  staff  member  needs  to  know  how  his  or  her  job  responsibilities  are  performed  using 
the  digital  equipment.  This  will  include  not  only  basic  instruction  on  button-pushing  and 
system  capabilities,  but  some  training  in  techniques  and  procedures  for  using  the  system 
efficiently. 

•  Tactical  skills  training :  This  phase  is  more  advanced  than  system  operations.  It  requires 
that  staff  members  begin  to  use  their  new  capabilities  to  perform  clearly  delineated  parts 
of  their  jobs.  Building  on  the  acquired  individual  skills,  this  training  should  be  collective, 
first  with  small  groups  of  staff  members  on  small  segments  of  tasks.  The  training  would 
gradually  build  up  to  having  the  full  staff  performing  in  an  integrated  and  self-directed 
way,  responding  to  job  situational  cues  and  actually  performing  the  appropriate  actions  in 
a  realistic  environment. 

•  Team  process  skills  training :  The  team  training  should  include  specific  information, 
practice,  and  feedback  addressing  decision-making,  team  skills,  situational  awareness, 
and  information  management.  These  are  not  independent  in  actual  performance,  and 
should  not  be  artificially  separated  in  training.  Additionally,  they  should  not  be  separated 
from  the  job  situation,  as  generic  skills,  but  rather  should  be  practiced  and  discussed  as  an 
integral  part  of  the  mission  planning,  preparation,  and  execution  cycle. 

Training  Structure 

Review  of  the  literature  led  to  identification  of  four  instructional  concepts  that  were  expected  to 
aid  in  both  acquisition  and  retention  of  skills.- 

•  Advance  organizer:  The  initial  information  presentations  to  the  participants  should 
include  multiple  instances  of  advance  organizers  (Smith  et  al.,  1997;  Cannon-Bowers  et 
al.,  1992,  1998;  West  et  al.,  1991).  For  whatever  is  novel  within  the  future  environment, 
the  training  audience  should  be  specifically  instructed  on  its  relation  to  the  “old”  way  of 
doing  business,  and  told  why  the  change  has  been  made.  If  there  are  elements  of  the  new 
environment  that  are  unchanged,  these  should  also  be  pointed  out  to  the  training 
audience.  Job  aids  that  graphically  show  the  old,  the  new,  and  the  linkage  should  be  used 
where  possible. 

•  Part-task  training:  Incorporating  principles  of  part-task  training  means  that  the  system 
training,  tactical  skills  training,  and  team  skills  training  will  each  be  broken  down  into 
simple  chunks  (Mane  et  al.,  1989;  Stammers,  1992;  Means  et  al.,  1993).  Training 
participants  will  receive  training  on  small  segments  of  tasks  and  may  even  be  clustered 
into  small  groups  to  receive  specific  system  training.  As  the  training  proceeds,  the  task 
segments  will  be  aggregated  and  the  small  groups  brought  together  to  allow  fully 
integrated  practice  of  tasks  and  activities. 
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~  The  structure  described  here  is  very  similar  to  that  of  the  structured  simulation-based  training  that  ARI  has  been 
developing  since  1993  (Campbell  et  al.,  1995;  Campbell  et  al.,  1997;  Campbell  et  al.,  1998  ). 


•  Deliberate  practice :  As  the  part-tasks  are  trained,  there  should  be  numerous  planned 
opportunities  for  deliberate  practice  of  the  skills  just  taught  (Frederiksen  &  White,  1989; 
Ericsson  et  al.,  1993;  Means  et  al.,  1993,  Jones,  1989).  This  practice  time  must  be 
structured  to  focus  on  the  skills  and  integration  of  previously  taught  skills,  and  should 
focus  on  training  to  criterion.  That  is,  not  only  should  ample  time  be  provided,  but 
participants  should  demonstrate  their  grasp  of  the  skills  several  times  (thus  avoiding 
lucky  guesses  and  allowing  for  overpractice)  before  moving  on  to  the  next  segment. 
Feedback  is  a  necessary  component  of  deliberate  practice;  participants  should  not  have  to 
decide  for  themselves  whether  or  not  they  have  achieved  proficiency. 

•  Context-based,'.  Finally,  all  of  the  training,  from  the  basic  and  advanced  system  training  to 
the  fully  integrated  full  staff  exercises,  should  be  presented  in  the  context  of  realistic  job 
requirements  (Means  et  al.,  1993;  Salisbury,  1990;  Cannon-Bowers  &  Bell,  1997; 
Cannon-Bowers  et  al.,  1992).  This  can  be  accomplished  even  for  the  simplest  task 
segments  in  the  system  training,  by  telling  or  showing  the  participants  how  the  skill  will 
be  used  as  they  perform  their  jobs.  The  complex  integrated  exercises  that  come  toward 
the  end  of  training  can  initially  have  simple  scenarios,  with  later  exercises  providing 
greater  challenges  by  means  of  very  high-intensity  scenarios  (e.g.,  less  time,  more  enemy, 
conflicting  information,  degraded  conditions). 

Training  Conditions 

Although  the  training  approach  described  above  is  general  enough  to  be  used  for  future  staff 
training  under  a  range  of  conditions,  there  are  four  underlying  assumptions. 

•  First,  the  training  is  intended  for  staffs  who  are  already  moderately  proficient  in  their 
individual  skills  and  who  are  familiar  with  basic  staff  operations.  They  need  not  be  highly 
experienced,  nor  is  it  necessary  that  they  have  worked  together  as  a  staff  for  any  great 
length  of  time. 

•  Second,  it  is  assumed  that  the  training  participants  are  not  yet  familiar  with  most  of  the 
key  elements  of  their  “new”  environment.  That  environment  may  involve,  for  example,  a 
new  type  of  staff  organization,  a  new  command  post  setup,  or  new  digital  capabilities. 
The  training  is  designed  to  cover  the  novel  elements  and  also  to  help  the  staff  leverage 
those  elements  in  order  to  increase  their  proficiency. 

•  Third,  it  is  assumed  that  they  are  operating  in  an  information-rich  environment.  The 
training  approach  is  specifically  aimed  at  operating  environments  wherein  staff  members 
must  deal  with  complex  information  management  and  communication  issues.  This 
assumption  is  not  a  necessary  one;  the  procedures  and  techniques  should  also  be  useful  in 
conventional  (analog)  environments,  but  the  need  for  the  specific  communication  and 
information  management  skills  is  not  as  obvious. 

•  Fourth,  it  is  assumed  that  some  type  of  virtual  or  constructive  simulation  will  be  used. 
Although  this  is  not  a  necessary  condition,  simulation  is  very  useful  in  supporting 
deliberate  practice  and  providing  realistic  contexts. 


Approach  for  BCR  III  Training  and  Assessment 


As  presented  in  the  Battle  Lab  Experiment  Plan  (BLEP)  for  the  BCR  III  (MMBL,  1998),  the 
purpose  of  the  experiment  was  to  examine  advanced  digitization’s  effects  on  battle  command  at 
brigade  and  below  .  The  central  issue  for  the  experiment  was:  If  the  Commander  is  provided 
timely,  accurate,  decision-centered  information  that  allows  him  to  visualize  the  battlefield,  how 
is  the  Commander’s  ability  to  conduct  battle  command  improved?  The  BCR  III  was  constructed 
to  allow  the  MMBL  to  focus  on  these  issues  by  putting  an  intact  squadron-level  unit  into  a  future 
digital  environment,  presenting  mission  situations,  and  observing  and  collecting  data  on  how  the 
unit  performed.  It  was  known  from  the  results  of  the  earlier  BCRs  that  participants  needed 
training  on  operating  the  digital  system  and  operating  within  a  reengineered  staff  structure.  An 
intact  squadron  staff  was  scheduled  to  participate  in  the  BCR  III,  so  that  there  would  not  be  the 
problem  of  providing  entry-level  staff  training. 

BCR  Training  Environment 

An  enhanced  Modular  Semi-Automated  Forces  (ModSAF)  system  was  used  to  emulate  digital 
C4I  capabilities  at  each  workstation.  The  advanced  C4I  systems  emulated  in  ModSAF  were 
referred  to  collectively  as  the  surrogate  command,  control,  communications,  and  computers 
(SC4)  system.  They  replicated  a  wide  range  of  capabilities,  including: 

•  Battle  commander’s  display,  represented  by  the  ModSAF  plan  view  display  (PVD).  On 
the  PVD,  commanders  and  other  staff  membersare  able  to  view  movements  of  all  of  their 
own  systems,  as  well  as  any  opposing  forces  (OPFOR)  units  detected  by  satellite  or  other 
sensors.  Overlays  can  be  drawn  on  the  PVD,  users  can  add  labels  or  other  notes,  and 
there  are  tools  that  show  past  events  and  project  future  movements. 

•  Stealth  display,  providing  a  360°  view  of  the  battlefield  and  all  of  the  systems  that  are 
visible  on  the  PVD  (i.e.,  friendly  and  detected  OPFOR). 

•  Video  teleconference  (VTC)  linking  the  Commander  and  his  staff. 

•  Collaborative  whiteboard  capability,  to  allow  the  Commander  to  present  his  intent  and 
guidance  to  the  staff  visually  and  quickly.  Whiteboard  conference  participants  can  upload 
images  from  their  own  PVDs,  draw  in  different  colors  on  those  images,  add  clipart-style 
labels  and  icons,  and  type  words  onto  the  whiteboard. 

Vehicles  and  weapon  systems  were  represented  in  either  constructive  or  virtual  simulation. 
Constructive  simulation  (ModSAF)  was  used  to  generate  and  control  the  OPFOR,  friendly  forces 
below  the  company  level,  and  unmanned  vehicles  replicating  both  aerial  and  ground  sensors 
(referred  to  as  UAVs  and  UGVs,  respectively).  Constructive  simulation  workstations  were  used 
by  the  fire  support  element  commander,  forward  support  company  commander,  two  of  the 
maneuver  company  commanders,  and  the  three  platoon  leaders  of  another  maneuver  company. 

In  the  virtual  environment,  simulators  were  used  to  represent  several  vehicles.  These  included 
the  Commander  and  deputy  commander  vehicles,  represented  by  reconfigurable  simulators;  staff 
operations  vehicles  (SOVs),  represented  by  command  and  control  vehicle  (C2V)  mockups;  and 


3  Information  for  this  section  of  the  report  has  been  adapted  from  the  BLEP  (MMBL,  1998). 


scout  vehicles,  represented  by  Future  Scout/Cavalry  System  mockups.  The  virtual  and 
constructive  environments  were  linked  by  means  of  distributed  interactive  simulation  (DIS)  to 
form  the  seamless  battlefield  environment  for  the  participants. 

Experiment  Participants  and  Organization 

The  primary  training  audience  (14  members)  comprised  the  Commander  and  selected  staff  of  an 
active  Army  cavalry  squadron.  The  primary  training  audience  was  augmented  by  an  extended 
training  audience  (21  members),  which  included  three  maneuver  company  commanders  and  their 
deputies,  three  maneuver  platoon  leaders  and  one  mortar  platoon  leader  for  one  of  the 
companies,  a  battery  commander  and  his  deputy,  a  forward  support  company  commander  with 
his  deputy,  and  the  scout  platoon  leader  and  platoon  sergeant.  A  total  of  35  soldiers  from  the 
squadron  participated  in  the  experiment  as  members  of  the  primary  training  audience  and 
extended  training  audience. 

One  major  experimental  feature  for  the  BCR  III  was  the  reengineered  battalion4  command  group 
structure  for  the  primary  training  audience,  depicted  in  Figure  1.  It  comprised  four  staff  nodes: 
the  command  group  node  (Command  1),  the  deputy  commander  node  (Command  2),  the  current 
operations  node  (Control  1),  and  the  future  operations  node  (Control  2).  Each  node  was  staffed 
by  3  or  4  primary  staff  members. 


Command  1 

Command  2 

Control  1 

Control  2 

(Command  Group) 

Deputy  Commander 

Battle  Captain 

Battle  Captain 

Commander 

Operations  Officer 

Friendly  Operations 

Friendly  Operations 

Effects  Operations 

Operations  NCO 

Enemy  Operations 

Enemy  Operations 

Enemy  Operations 

Sensor  NCO 

Sensor  NCO 

Figure  1.  Reengineered  battalion  command  group  structure. 

The  BCR  III  training  audience  began  with  this  structure,  and  was  given  a  minimum  of 
information  about  the  roles  and  responsibilities  of  each  staff  member  in  each  node.  As  the 
experiment  progressed,  the  Commander  was  encouraged  to  realign  roles  and  responsibilities  in 
order  to  explore  different  structures. 

MMBL  personnel  served  as  research  assistants  (RAs)  who  conducted  the  training  and  mentored 
the  training  audience  as  they  operated  the  SC4  system.  Other  MMBL  personnel  staffed  the 
higher  headquarters  response  cell  (roleplaying  the  brigade  staff)  as  well  as  the  constructive 
simulation  workstations  for  combat  support,  combat  service  support,  and  the  OPFOR. 


4  A  note  on  terminology:  The  BCR  was  intended  to  address  staff  organization  and  C4I  use  in  the  context  of  an  armor 
divisional  battalion.  The  unit  supporting  the  BCR  was  a  squadron  drawn  from  a  cavalry  regiment.  The  echelons  are 
roughly  equivalent  within  their  respective  structures. 


Training  Design  and  Development  Process 


The  process  for  development  of  the  training  followed  the  Army’s  systems  approach  to  training 
(Department  of  the  Army  [DA],  1999)  and  the  procedures  outlined  in  Campbell  et  al.  (1997). 
The  development  process  always  begins  with  a  thorough  front-end  analysis  of  what  is  known  and 
what  needs  to  be  learned  about  the  training  needs,  training  conditions,  and  training  audience. 
Analysis  then  leads  into  design  and  development,  resulting  in  products  that  should  be  appropriate 
for  the  conditions  imposed  by  the  environment  and  that  should  satisfy  the  training  requirement. 
The  design  and  development  work  incorporates  the  training  content  and  structure  decisions 
described  above,  and  includes  attention  to  measurement  for  feedback  and  program  evaluation. 

Throughout  the  analysis,  design,  and  development  activities,  training  products  are  under  constant 
review  and  are  revised  as  necessary.  During  initial  implementation,  formative  evaluation  of  the 
training  products  and  process  is  conducted  to  assess  success  in  meeting  the  training 
requirements.  Measurement  should  continue  for  additional  implementations  of  the  training,  so 
that  developers  can  track  performance  deficiencies  that  should  be  addressed  in  the  training. 

BCR  III  Training  Analysis 

The  analysis  process  underlying  design  of  the  BCR  III  train-up  included: 

•  defining  the  battalion  staff  processes  in  terms  of  responsibilities  for  nodes  and  for 
individuals  within  nodes; 

•  defining  how  the  battalion  staff  processes  should  (or  could)  be  performed  within  the 
nodes; 

•  defining  what  the  SC4  tools  are  and  what  they  can  do;  and 

•  defining  individual  functions  and  tasks  using  the  SC4  tools. 

The  analysis  served  as  the  foundation  for  design  of  the  training  that  focused  on  staff 
performance.  The  design  work  on  the  project  included  making  recommendations  for  basic  and 
advanced  SC4  training,  as  well  as  staff  process  training. 

Staff  Training  Program 

The  approach  was  hierarchical,  outlining  a  5-level  training  sequence.  It  assumed  (in  accordance 
with  the  assumptions  described  earlier)  that  participants  would  have  some  experience  working 
together  as  a  team.  The  training  structure  proceeded  through  orientation,  individual  training, 
small  group  training,  and  a  series  of  tactical  decision-making  exercises  (TDXs).  The  purpose  of 
each  level  is  described  below. 

Level  1:  Initial  Orientation.  In  Level  1,  the  training  participants  began  with  an  introduction  and 
orientation  to  the  train-up  and  the  experiment  itself.  The  Level  1  session  was  designed  to  serve  as 
an  advance  organizer,  in  accordance  with  research  indicating  that  learners  do  better  if  they  are 
told  what  they  will  be  learning,  how  they  will  receive  it,  and  how  it  relates  to  current  knowledge 
and  processes.  The  orientation  introduced  several  novel  aspects  of  how  they  would  operate:  the 


4-node  staff  structure,  the  SC4  system  capabilities,  their  own  combat  and  combat  support 
resources,  and  the  OPFOR  capabilities. 

Level  2:  Fundamentals.  The  fundamentals  training,  Level  2,  was  designed  to  provide  SC4 
equipment  familiarity  combined  with  general  system  operations.  At  this  stage  of  training,  the 
information  was  basic  and  focused  on  the  individual  with  a  limited  context.  Each  participant 
received  both  instruction  and  guided  practice. 

Level  3:  Functions.  The  Level  3  training  was  then  designed  to  provide  the  context-based 
training  by  clustering  the  system  tools  into  functional  groups  (e.g.,  create  overlays,  conduct  a 
whiteboard  teleconference).  The  groupings  for  the  Level  3  training  were  a  result  of  the  task,  tool, 
and  functions  analysis  discussed  in  the  previous  section.  Though  this  training  still  focused 
primarily  on  the  individual,  it  grouped  together  individuals  from  different  nodes  who  performed 
similar  tasks.  This  arrangement  provided  unit  members  the  opportunity  to  discuss  how  they 
planned  to  use  the  system  in  the  context  of  their  jobs  in  the  different  nodes.  Level  3  training  was 
expected  to  be  lengthy,  requiring  4-6  hours.  In  order  to  provide  deliberate  practice  opportunities, 
Levels  2  and  3  training  included  structured  practice  exercises  that  tracked  directly  with  the 
material  covered  in  the  training  and  could  serve  as  checks  on  whether  the  staff  had  acquired  the 
skills  taught. 

Level  4:  Tasks  Training  and  Practice.  The  Level  4  training  was  a  series  of  small  group 
structured  exercises  that  provided  the  first  opportunity  to  practice  the  new  skills  in  the  context  of 
brief  scenarios.  It  was  designed  and  developed  to  provide  the  transition  from  system  training  to 
staff  process  training.  The  focus  of  the  Level  4  exercises  was  carried  another  step  beyond  the 
Level  3  training,  by  prompting  the  unit  members  to  think  about  how  they  were  to  work  together 
within  their  node.  No  new  system  tools  or  functions  were  introduced  during  this  training. 

The  exercises  were  structured  so  that  each  node  would  train  independently  on  the  same  collective 
function.  This  structure  allowed  the  unit  members  to  further  explore  and  understand  their  roles 
and  functions  within  each  node,  free  of  outside  distractions.  It  also  reinforced  the  multi¬ 
functionality  aspect  required  to  operate  in  the  experimental  staff  structure.  These  exercises  were 
to  prepare  the  unit  members  to  participate  in  the  Level  5  TDXs,  which  focused  on  collective 
training. 

Level  5:  Collective  Training.  Linally,  Level  5  training  consisted  of  four  TDXs  and  embedded 
team  training  sessions.  The  TDXs  brought  together  all  BCR  III  training  participants  (e.g., 
primary  training  audience,  company  commanders,  scouts)  to  conduct  collective  structured 
exercises  using  the  full  capabilities  of  the  SC4  system.  The  exercises  provided  the  opportunity  to 
practice  new  skills  in  a  tactical  scenario  similar  to  the  type  of  mission  used  during  the  pilot  and 
experiment  trials.  These  exercises  not  only  allowed  practice  on  the  system,  but  provided  the 
context  for  introduction  of  training  sessions  addressing  decision-making  and  team  process  skills. 
Design  of  the  TDXs  was  based  on  the  performance  analysis  of  the  full  BCR  II  mission  flow 
described  earlier,  from  planning  and  preparation  through  execution  and  sustainment  activities. 
This  analysis  led  to  the  delineation  of  four  distinct  segments  that,  together,  required  the 
participants  to  practice  a  wide  range  of  staff  processes  and  SC4  functions.  The  four  segments 


were:  a)  mission  analysis  with  wargaming;  b)  mission  rehearsal;  c)  execution  of  a  squadron 
branch;  and  d)  execution  of  a  brigade  sequel. 

Level  4  and  5:  Team  Training  Sessions.  Team  training  sessions  were  included  before  and  after 
Level  4  training  and  as  an  integral  part  of  the  Level  5  TDXs.  These  sessions  were  designed  to 
help  the  staff  to  focus  on  various  aspects  of  performance  in  digital  environments,  and  also  to 
provide  them  with  techniques  for  continuing  to  examine  and  improve  their  performance.  Five 
types  of  sessions  were  presented: 

•  Information  Management  ( Level  4):  Information  session  to  focus  attention  on  the 
increased  information  management  load  imposed  by  the  digital  environment.  Included  a 
generic  “information  management”  exercise  to  highlight  difficulties;  presentation  of 
required  knowledge  for  information  management,  model  of  information  management, 
and  common  errors  in  information  management;  and  discussion  of  low  level  strategies 
and  solutions  (e.g.,  e-mail  formatting  protocols)  and  high  level  strategies  and  solutions 
(e.g.,  understanding  Commander’s  intent). 

•  Roles  and  Functions  ( Level  4):  Information  session  to  encourage  the  staff  to  work  out 
their  individual  and  node  roles.  Included  a  short  presentation  on  the  importance  of  roles 
and  functions;  description  of  the  connections  between  roles  and  functions  and  decision¬ 
making,  coordination,  and  information  management;  discussion  of  how  the  unit  members 
perceive  their  roles  and  functions  for  the  upcoming  BCR  III.  Also  presented  a  format  for 
the  unit  to  use  to  self-correct  during  the  exercise  as  needed. 

•  Pre-Action  Analysis  ( Level  5):  Conducted  before  each  TDX  to  focus  staff  attention  on 
shared  understandings  of  roles  and  functions,  information  management,  workload,  and 
situational  awareness.  The  purpose  is  to  anticipate  potential  problem  areas  and  help 
generate  how  they  will  solve  problems  should  they  arise.  These  sessions  will  also  allow 
time  to  review  those  action  items  identified  in  previous  debriefs  for  sustainment  or 
improvement. 

•  Commander’s  Timeout  ( Level  5):  Recalibrate  the  staff  to  the  Commander’s  situational 
awareness,  at  least  once  during  each  TDX.  The  benefits  of  performing  a  Commander’s 
Timeout  are  that  it  causes  the  Commander  to  stop  and  think  about  his  situational 
awareness  and  to  communicate  that  to  the  staff.  It  allows  the  Commander  to  gain 
information  to  questions  he  may  have  and  help  the  team  to  focus  on  keeping  the 
Commander  aware  of  the  ground  truth. 

•  Team  Decision-Making  Debrief  ( Level  5):  Focus  on  issues  related  to  team  processes, 
decision-making,  roles  and  functions,  and  information  management.  Conducted  at  the 
end  of  each  TDX.  The  Commander  and  staff  will  also  identify  action  items  for 
sustainment  and  improvement. 

The  training  products  developed  for  the  basic  and  advanced  SC4  training  included  training  plan 
outlines,  which  were  turned  over  to  the  MMBL  for  completion  of  the  structured  practice 
exercises.  For  staff  process  training,  researchers  developed  and  produced  the  TSPs  that  included 
tactical  materials,  observation  and  feedback  mechanisms  and  aids,  and  simulation  files  (as 
prescribed  in  the  U.S.  Army  Training  and  Doctrine  Command  [TRADOC]  Regulation  350-70 
[DA,  1999]).  The  major  research  products  associated  with  the  prototype  training  for  the  BCR  III 


implementation  are  described  in  the  project  final  report  (Throne  et  al.,  1999)  and  presented  in  the 
set  of  materials  entitled  Training  and  Measurement  Support  Package  for  Battle  Command 
Reengineering  111  (MMBL,  1999). 

The  period  of  time  provided  for  “train  up”  was  five  days.  Level  2  and  3  training  focused  on  the 
individual,  Level  4  progressed  to  a  small  group  (node)  focus,  and  the  Level  5  TDXs  and  team 
training  sessions  brought  all  elements  together  for  collective  training. 

Formative  Evaluation 

The  effectiveness  of  the  training  was  evaluated  during  implementation  in  the  BCR  III,  mostly 
through  the  use  of  surveys  and  interviews.  The  issues  addressed  during  the  training  evaluation 
were  the  training  program  structure,  support  materials,  and  content.  The  training  surveys  were 
filled  out  by  all  the  training  participants  at  the  end  of  each  level  of  training  and  on  the  last  day  of 
the  experiment.  Interviews  were  conducted  on  the  last  day  of  training  with  the  Commander  and 
certain  RAs  and  observers,  and  on  the  last  day  of  the  experiment  with  selected  primary  training 
audience  members  and  observers. 


Results  and  Discussion 

Results  pertaining  to  the  training  structure,  the  support  materials,  and  the  content  of  the  training 
are  discussed  below.  While  the  training  and  experiment  phases  were  generally  implemented  as 
outlined  in  the  BLEP,  there  were  some  conditions  of  the  experiment  environment  and  staffing 
that  impacted  on  the  training  program  and  on  data  collection  efforts.  These  implementation 
conditions  and  their  impact  on  training  are  identified  in  the  discussion  below. 

Participant  reaction  to  Training  Structure 

Structure  includes  the  training  flow,  implementation  model,  training  scenarios,  and  support 
materials.  Overall,  the  participants  found  the  training  to  be  adequate  in  terms  of  sequencing  and 
amount  of  instruction,  coaching,  and  practice.  However,  their  comments  on  the  surveys  and 
interviews  identified  many  areas  for  improvement. 

The  progression  from  individual  to  small  group  to  collective  training  appears  to  be  acceptable. 
Respondents  who  recognized  the  progression  affirmed  it;  those  who  did  not  recognize  it 
requested  such  a  progression.  In  particular,  they  indicated  that  the  orientation  could  have  been 
more  useful  in  preparing  them  for  the  train-up  that  followed.  It  should  be  noted  that  the 
orientation  session  was  not  conducted  as  planned,  using  advance  organizer  principles.  Instead,  a 
general  overview  of  the  many  SC4  capabilities  was  presented.  Additionally,  the  Level  3  plan  for 
staff  groupings  to  reinforce  position-specific  procedures  was  disrupted  by  other  system  training 
unrelated  to  the  intended  functional  training. 

The  training  sequence  and  time  allocation  requires  some  examination.  The  initial  tools  training 
(Levels  2  and  3)  did  not  contain  enough  formal  instruction,  practice,  or  skill  assurance.  In  fact, 
most  of  the  designed  structured  practice  exercises  were  not  implemented.  As  a  result,  during 
small  group  training  (Level  4),  individuals  were  still  attempting  to  learn  individual  skills,  and  the 


planned  multiechelon  training  was  frustrating  for  participants  at  both  the  squadron  and  company 
levels. 


The  TDXs  were  seen  as  an  important  component  of  preparation.  While  the  mission  segments 
(planning,  rehearsal,  and  execution)  were  acceptable,  the  scenarios  may  have  been  both  too  slow¬ 
paced  and  too  deliberate.  By  incorporating  more  high-stress  situations,  the  staff  would  have  had 
more  incentive  to  conduct  situational  awareness  recalibrations  and  intensive  decision-making 
debriefs.  Some  participants  also  suggested  including  more  TDXs  so  they  would  have  had  more 
opportunities  to  practice  and  refine  their  tactics,  techniques,  and  procedures  (TTPs). 

Implementation  of  the  Level  4  and  Level  5  staff  process  training  was  intended  to  be  conducted 
initially  by  the  MMBL  Training  Director  and  later  transitioned  to  the  Commander.  Because  the 
Training  Director’s  time  was  limited  during  the  week  prior  to  the  experiment,  efforts  to  prepare 
him  for  that  role  were  similarly  limited.  Time  with  the  Commander  was  even  more  constrained, 
with  the  result  that  he  was  unable  to  fully  implement  the  requirements. 

Utility  of  Training  Support  Materials 

The  support  materials  for  the  TDXs  (e.g.,  intelligence  summaries  [INTSUMs]  and  fragmentary 
orders  [FRAGOs])  were  seen  as  adequate.  The  single  item  most  often  noted  as  “missing”  or 
incomplete  during  the  train-up  period  was  a  clear  statement  of  the  training  objectives.  Revisions 
to  the  delivery  of  the  training  materials  must  include  this  simple  (and  embarrassingly  obvious) 
but  critical  feature.  Other  suggestions  pertaining  to  materials  focused  on  training  checklists  for 
the  RAs,  job  aids  showing  SC4  operations,  charts  of  blue  forces  (BLUFOR)  and  OPFOR  systems 
and  capabilities,  and  roadmaps  of  objectives  and  topics  for  the  training  week. 

Participant  Acceptance  of  Training  Content 

Results  concerning  the  training  content  centered  on  SC4  topics,  battlefield  environment 
information,  and  staff  process  training.  In  general,  participants  found  their  initial  training  to  be 
only  adequate  in  terms  of  what  they  learned  about  the  SC4  system.  Suggested  fixes  ranged  from 
more  time  for  structured  practice,  to  better-trained  RAs,  to  detailed  job  aids.  In  addition, 
inclusion  of  combat  service  support  (CSS)  functions  was  generally  judged  inadequate. 

One  of  the  assumptions  for  the  training  was  that  the  staff  members  would  be  familiar  with  their 
duties  and  responsibilities  in  a  conventional  staff  configuration.  Due  to  normal  turbulence  within 
the  unit,  however,  many  of  the  staff  members  were  relative  novices  in  staff  operations.  This 
meant  that  some  of  the  training  was  too  intensive,  attempting  to  build  on  skills  that  the 
individuals  were  lacking. 

Reactions  to  the  team  training  sessions  were  mixed.  Most  of  the  respondents  indicated  that  this 
staff  did  not  need  information  management  training  as  provided,  although  there  was  no  argument 
that  it  was  an  important  topic.  Nonetheless,  they  continued  to  have  information  management 
difficulties  throughout  the  TDXs  and  into  the  experiment.  The  timing  of  the  information 
management  session,  coming  as  it  did  prior  to  any  significant  information  flow,  may  have  been 
the  problem. 


Likewise,  most  participants  agreed  that  the  Pre-Action  Analysis,  Commander’s  Timeout,  and 
Team  Decision-Making  Debrief  were  valuable  in  concept  but  not  particularly  useful  as 
implemented.  Whether  this  is  due  to  the  way  in  which  they  were  conducted,  or  to  the  way  they 
were  designed  is  unknown.  However,  the  concepts  should  be  considered  for  future  use. 

Lessons  Learned  for  Future  Research 

These  lessons  are  addressed  to  several  audiences,  including  developers  of  future  staff  training 
and  researchers  conducting  future  systems  experiments.  The  lessons  fall  into  three  categories: 

•  Training  structure 

•  Training  support  materials 

•  Training  content  (specifically,  team  and  decision  skills  training). 

Lessons  Learned  on  Training  Structure 

The  training  prototype  was  designed  as  a  hierarchical  progression  from  an  initial  orientation 
through  individual  and  small  group  training  to  collective  training  including  team  process  training 
sessions.  It  incorporated  features  recommended  in  the  training  literature,  including  an  advance 
organizer,  a  realistic  underlying  context,  and  both  formal  instruction  and  hands-on  practice 
opportunities.  All  of  these  features  should  be  retained. 

However,  the  features  must  be  explicitly  implemented  in  accordance  with  the  design  plan.  Some 
of  the  ways  of  ensuring  that  these  features  are  advantageously  used  include: 

•  Unit  preparation  plan:  In  advance  of  the  training,  preparation  materials  for  the  unit  could 
help  the  unit  to  derive  greater  benefit  from  the  training  itself.  A  combination  of  listening 
(e.g.,  a  briefing  or  videotape),  reading  (e.g.,  background  materials,  explanations), 
showing  (e.g.,  what  the  systems  look  like,  how  to  use  them,  what  they  can  do  for  the 
user),  and  doing  (e.g.,  preparing  tactical  products,  refining  the  unit  standing  operating 
procedure  [SOP]  and  TTPs)  should  be  incorporated. 

•  Organization  and  content  of  the  initial  orientation:  The  advance  organizer,  as  a  way  of 
introducing  a  novel  situation,  holds  great  promise.  The  initial  orientation  should  be  as 
concise  and  direct  as  possible,  providing  necessary  information  to  get  participants  into 
their  first  day  of  training  with  an  understanding  of  what  they  will  be  doing. 

•  Time  allotted  to  different  phases  of  the  training:  Training  on  new  systems  should  include 
generous  amounts  of  time  for  hands-on  practice  and  exploration.  If  sufficient  time  is 
allowed  for  individual  practice  during  the  initial  learning  phase,  then  the  time  set  aside 
for  small  group  and  collective  practice  on  more  advanced  skills  can  be  used  more 
productively,  and  overall  time  may  even  be  reduced. 

•  Sequence  of  the  different  phases  of  the  training:  In  learning  a  novel  and  complex  system, 
a  series  of  training  cycles,  each  focused  on  a  subset  of  tools  and  functions  and  each 
including  the  individual  to  small  group  to  collective  progression,  will  serve  to  provide  the 
big  picture  context  more  often  and  more  effectively. 


•  Training  audience  members:  Training  does  not  just  happen  as  a  result  of  putting  people  in 
front  of  systems,  even  with  knowledgeable  assistants  on  hand.  For  any  new  system 
training,  providing  training  for  all  segments  of  the  training  audience  is  essential. 

Lessons  Learned  on  Training  Support  Materials 

The  lessons  learned  concerning  the  prototype  materials  are  generalizable  for  other  future  staff 
training.  Lessons  pertaining  specifically  to  the  team  training  sessions  are  reserved  to  the  next 
section. 


•  Creating  training  scenario  materials:  In  constructive  simulations,  much  of  the  scenario 
information  must  be  presented  by  the  simulation.  Developers  must  begin  working  in  their 
target  environment  quite  early,  in  order  to  ensure  that  capabilities  are  understood  and 
used.  For  the  BCR,  changes  to  the  SC4  systems  were  still  being  made  several  days  prior 
to  the  training  period,  making  inclusion  of  appropriate  features  and  creation  of  simulation 
files  difficult  at  best. 

•  Use  of  job  aids:  For  large  amounts  of  relatively  simple  information,  or  small  bytes  of 
obscure  information,  job  aids  should  be  provided. 

•  Training  audience  and  train-the-trainer  materials:  Without  complete  training  support 
packages,  information  concerning  the  training  objectives,  sequence  and  structure,  and 
methods  for  implementation  may  be  possible  by  means  of  memos  and  discussions,  but  it 
will  be  unreliable  at  best.  This  lesson  serves  in  a  small  way  to  validate  the  guidance 
contained  in  TRADOC  Regulation  350-70  (DA,  1999)  and  in  ARI  development 
methodologies  (Campbell  et  al.,  1995;  Campbell  et  al.,  1997;  Campbell  et  al.,  1998). 

•  Pilot  testing:  Full  pilot  testing  of  materials  prior  to  use,  in  the  form  of  a  dress  rehearsal,  is 
essential  (and  was  not  possible  for  the  BCR).  Otherwise,  the  first  operational  use  is  the 
pilot  test,  and  there  are  no  opportunities  to  refine  materials  before  use. 

Lessons  Learned  on  Training  Content  -T earn  and  Decision  Skills  Training 

The  prototype  training  served  as  a  very  rough  pilot  implementation  for  five  types  of  team 
training:  Information  Management,  Roles  and  Functions,  Pre-Action  Analysis,  Commander’s 
Timeout,  and  Tactical  Decision-Making  Debrief.  But  because  the  actual  implementation  of  the 
team  training  sessions  did  not  represent  the  designed  process,  assessment  of  the  outcome  of  the 
team  training  sessions  is  not  appropriate. 

Nonetheless,  a  great  deal  about  the  potential  for  incorporating  this  type  of  session  in  future  staff 
training  (future  training  as  well  as  training  for  future  staffs)  has  been  learned.  Comments 
received  from  the  training  audience  members  indicate  that  the  content  of  the  sessions  had 
potential  for  being  useful,  and  that  sessions  conducted  as  part  of  collective  training  exercises 
would  be  valuable.  Specific  suggestions  for  future  training  are  discussed  below. 

•  Unit  leadership  preparation:  This  issue  is  closely  related  to  one  discussed  earlier  on  unit 
preparation,  but  is  specifically  targeted  to  preparation  of  the  unit  leader.  This  individual  is 
the  single  most  important  influence  on  the  success  of  a  training  program.  If  the  leader  is 
educated  in  advance  as  to  the  purposes  and  processes  in  the  training,  and  is  persuaded  to 


wholeheartedly  support  those  purposes  and  processes,  then  the  training  program  has 
every  opportunity  for  success. 

•  Information  management  training:  Given  the  increased  amounts  of  information  available 
to  all  staff  members,  information  management  should  be  a  ripe  topic  for  training. 
Developers  still  need  to  research  the  current  information  management  training  that  is 
provided,  the  specific  difficulties  presented  in  the  environment  for  the  training,  and  the 
techniques  and  procedures  that  will  be  useful  in  that  environment. 

•  Presentation  of  roles  and  functions  session:  The  roles  and  functions  session  has  potential 
to  be  one  of  the  most  easily  implemented  and  directly  beneficial  sessions  for  team 
training.  These  sessions  need  to  be  conducted  frequently,  with  discussions  specific  to  the 
current  activity  or  mission  plan  (rather  than  generic  “Who  am  I?”  and  “What  do  I  do?” 
discussions). 

•  Facilitation  of  the  pre-action  analysis,  timeout,  and  debrief:  In  order  for  the  Commander 
to  be  a  part  of  the  training  audience,  the  role  of  training  must  be  taken  by  someone 
outside:  an  observer/controller  (O/C),  Training  Director,  or  representative  of  the  higher 
headquarters.  This  individual  needs  subject  matter  expertise  and  credibility,  training 
materials,  instruction  and  ongoing  guidance,  and  the  authority  to  lead  such  sessions. 

Summary 

The  BCR  III  experiment  provided  a  unique  opportunity  to  implement  prototype  staff  training 
methods  for  future  forces.  The  prototype’s  topics  and  structure  and  the  methodology  for 
development  are  grounded  in  the  research  literature.  Development  of  the  prototype  training 
system  followed  a  tried  and  proven  procedure  laid  out  by  TRADOC  and  in  previous  ARI 
projects.  Difficulties  in  implementation  generally  occurred  where  the  procedures  had  not  been 
strictly  followed.  However,  the  formative  evaluation  provided  valuable  information  for  program 
revisions  in  preparation  for  other  experimentation. 

Additional  trials  of  the  prototype  training  and  evaluation  systems  are  required  to  validate  its 
efficacy  and  utility.  As  capabilities  of  advanced  C4I  systems  are  enhanced,  future 
experimentation  should  lead  to  further  development  of  a  prototype  training  method  (particularly 
in  staff  training  strategies)  that  targets  higher-order  cognitive  skills  needed  on  the  digital 
battlefield.  Advances  in  training  need  to  parallel  technological  advances  in  the  information  age 
before  the  digital  Army  of  Force  XXI  can  maintain  battlefield  dominance. 

References 

Alberts,  D.  S.  (1996).  The  unintended  consequences  of  information.  The  Center  of  Advanced 
Concepts  and  Technology,  Washington,  DC:  National  Defense  University. 

Campbell,  C.  H.,  Campbell,  R.  C.,  Sanders,  J.  J.,  Flynn,  M.  R.,  &  Myers,  W.  E.  (1995). 

Methodology  for  the  development  of  structured  simulation-based  training  (ARI 
Research  Product  95-08).  Alexandria,  VA:  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 


Campbell,  C.  H.,  Deter,  D.  E.,  &  Quinkert,  K.  A.  (1997).  Report  on  the  expanded  methodology 
for  development  of  structured  simulation-based  training  programs  (ARI  Research 
Report  1710).  Alexandria,  VA:  U.  S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

Campbell,  C.  H.,  Ford,  L.  A.,  Campbell,  R.  C.,  &  Quinkert,  K.  A.  (1998).  A  procedure  for 
development  of  structured  vignette  training  exercises  for  small  groups  (ARI 
Research  Product  98-37).  Alexandria,  VA:  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 

Cannon-Bowers,  J.  A.,  &  Bell,  H.  H.  (1997).  Training  decision  makers  for  complex 

environments:  Implications  of  the  naturalistic  decision  making  perspective.  In  C. 

E.  Zsambok  &  G.  Klein  (Eds.),  Naturalistic  decision  making  (pp.  99-110). 
Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  Inc. 

Cannon-Bowers,  J.  A.,  Rhodenizer,  L.,  Salas,  E.,  &  Bowers,  C.  A.  (1998).  A  framework  for 
understanding  pre-practice  conditions  and  their  impact  on  learning.  Personnel 
Psychology ,  51,  291-320. 

Cannon-Bowers,  J.  A.,  Salas,  E.,  &  Converse,  S.  (1992).  Shared  mental  models  in  expert  team 
decision  making.  In  N.  J.  Castellan,  Jr.  (Ed.),  Current  issues  in  individual  and 
group  decision  making  (pp.  221-246).  Mahwah,  NJ:  Lawrence  Erlbaum 
Associates,  Inc. 

Department  of  the  Army.  (1999).  Training  development  management,  processes,  and  products 
(TRADOC  Regulation  350-70).  Fort  Monroe,  VA:  Headquarters,  U.  S.  Training 
and  Doctrine  Command. 

Elliott,  G.  S.  (1998).  System  evaluation  report  for  the  battle  command  reengineering  II  (BCR  II). 
U.S.  Army  operational  Test  and  Evaluations  Command  Test  and  Evaluation 
Coordination  Office  (TECO),  Fort  Knox,  Kentucky,  40121-5000. 

Elliott,  G.  S.,  Sterling,  B.  S.,  &  Lickteig,  C.  W.  (1998).  System  evaluation  report  for  taskforce 
tactical  operations  center  (TOC).  U.  S.  Army  Operational  Test  and  Evaluations 
Command  Test  and  Evaluation  Coordination  Office  (TECO)  Fort  Knox,  Kentucky, 
40121-5000. 

Ericsson,  K.  A.,  Krampe,  R.  T.,  &  Tesch-Romer,  C.  (1993).  The  role  of  deliberate  practice  in  the 
acquisition  of  expert  performance.  Psychological  Review,  100,  363-406. 

Frederiksen,  J.,  &  White,  B.  (1989).  An  approach  to  training  based  upon  principled  task 
decomposition.  Acta  Psychologica,  71,  89-146. 

Jones,  M.  B.  (1989).  Individual  differences  in  skill  retention.  American  Journal  of  Psychology, 
102,  183-196. 


Mane,  A.  M.,  Adams,  J.  A.,  &  Donchin,  E.  (1989).  Adaptive  and  part-whole  training  in  the 

acquisition  of  a  complex  perceptual-motor  skill.  Special  Volume:  The  Learning 
Strategies  Program:  An  Examination  of  the  Strategies  in  Skill  Acquisition.  Acta 
Psychologica,  71,  179-196. 

Means,  B.,  Salas,  E.,  Crandall,  B.,  &  Jacobs,  O.  T.  (1993).  Training  decision  makers  for  the  real 
world.  In  G.  A.  Klein,  J.  Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (Eds.), 
Decision  making  in  action:  Models  and  methods  (pp.  306-326).  Norwood,  NJ: 
Ablex  Publishing  Corporation. 

Mounted  Maneuver  Battlespace  Lab.  (1998).  Battle  lab  experiment  plan  (BLEP)  for  battle 
command  reengineering  Ill  (BCR  III).  (Available  from  Mounted  Maneuver 
Battlespace  Lab,  Attn:  ATZK-MW,  Fort  Knox,  KY  40121) 

Mounted  Maneuver  Battlespace  Lab.  (1999).  Training  and  Measurement  Support  Package,  Battle 
Command  Reengineering  III  (BCR  III),  Mounted  Maneuver  Battlespace  Lab, 
Volumes  1-5.  (Available  from  Mounted  Maneuver  Battlespace  Lab,  ATTN: 
ATZK-MW,  Fort  Knox,  KY  40121). 

Salisbury,  D.  F.  (1990).  Cognitive  psychology  and  its  implications  for  designing  drill  and 

practice  programs  for  computers.  Journal  of  Computer-Based  Instruction,  17,  23- 
30. 

Smith,  E.  M.,  Ford,  J.  K.,  &  Kozlowski,  S.  W.  J.  (1997).  Building  adaptive  expertise: 

Implications  for  training  design  strategies.  In  M.  A.  Quinones  &  A.  Ehrenstein 
(Eds.),  Training  for  a  rapidly  changing  workforce  (pp.  89-118).  Washington,  DC: 
American  Psychological  Association. 

Stammers,  R.  B.  (1982).  Part  and  whole  practice  training  for  procedural  tasks.  Human  Learning, 
1,  185-207. 

Throne,  M.  H.,  Holden,  W.  T.,  Deatz,  R.  C.,  Campbell,  C.  H.,  Sterling,  B.  S.,  &  Lickteig,  C.  W. 

(1999).  Prototype  staff  training  and  evaluation  methods  for  future  forces  (ARI 
Research  Report  1745).  Alexandria,  VA:  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 

West,  C.  K.,  Farmer,  J.  A.,  &  Wolff,  P.  M.  (1991).  Instructional  design:  Implications  from 
cognitive  science.  Englewood  Cliffs,  NJ:  Prentice  Hall. 


